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RESEARCH  PROJECT  ON  DECISION  ANALYTIC  TECHNOLOGY 


1.0  INTRODUCTION 


This  report  describes  a research  effort  extending  over 
two  and  a half  years  under  contract  N00014-75-C-0426 . The 
primary  objective  of  the  program  was  to  develop  the  technology 
in  which  decisions  and  inferences  are  made  by  individuals 
and  implemented  on  a computer.  The  conceptual  framework 
within  which  the  decision  aids  are  developed  is  personalist 
decision  theory  including  Bayesian  Inference. 

Specific  tasks  were  selected  to  take  advantage  of  the 
special  resources  of  Decisions  and  Designs,  Incorporated 
(DDI) , as  an  active  proponent  of  such  aids  in  defense  and 
government  applications. 

This  effort  parallels  two  other  research  programs 
monitored  by  the  Office  of  Naval  Research  and  conducted  by 
DDI:  "Research  on  Advanced  Decision  Technology"  funded  by 

ARPA,  and  "Application  of  Decision  Analysis  Technology  to 
Operational  Decision  Aiding  at  the  Task  Force  Command  Level" 
in  ONR's  Operational  Decision  Aids  Project. 

Tasks  were  undertaken  in  five  general  areas : method 
generalization,  development  of  specific  techniques,  develop- 
ment of  basic  theory,  behavioral  research,  and  illustrative 
case  studies.  They  resulted  in  eighteen  technical  reports, 
ten  working  papers,  and  seven  archival  publications  (see 
references) . Titles  and  abstracts  of  working  papers  are 
given  in  the  appendix.  The  following  section  outlines 
progress  achieved  on  each  of  the  primary  tasks. 
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2.0  MAIN  TASKS  AND  ACCOMPLISHMENTS 


2 . 1  Methodology  Generalization 

Substantial  effort  was  devoted  to  two  tasks  oriented 
toward  developing  the  "state  of  the  art"  of  decision  analysis 
at  a general  level.  These  tasks  were  designed  to  make 
available  to  users,  practitioners,  and  researchers  in  the 
field  of  decision  analysis  the  insights  and  generalized 
experience  acquired  by  DDI  staff  in  the  process  of  attacking 
a succession  of  real-world,  practical  problems. 

2.1.1  Taxonomy  matching  - The  first  task  was  to 
develop  a conceptual  taxonomic  framework  within  which  the 
"state  of  the  art"  of  applied  decision  analysis  could  be 
codified  for  the  purpose  of  matching  analytic  approaches  and 
techniques  to  different  kinds  of  decision  situations.  The 
results  of  this  effort  are  presented  in  a three-volume 
report  (see  references  11  & 12) . 

The  first  volume  describes  the  conceptual  fraune- 
work  within  which  experienced  decision  analysts  can  derive 
generalizations  and  communicate  them  to  decision  makers  and 
inexperienced  analysts.  The  fraunework  consists  of  a three- 
way  taxonomy:  decision  situations,  analytic  options,  and 
performance  measures. 

Within  this  framework,  a tentative  and  illustra- 
tive set  of  practical  guidelines  is  presented  to  help 
decision  analysis  users  and  practitioners  identify  appropriate 
analytic  approaches  for  any  given  decision  situation.  An 
attempt  is  made  to  suggest  a taxonomic  framework  for  codifying 
the  state  of  the  art  of  decision  analysis,  a language  for 
expressing  "matching  generalizations"  which  associate  the 
appropriate  analytic  option  to  a particular  situation.  This 
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language  has  three  main  components,  each  of  which  has  an 
exhaustive  numerical  coding  scheme. 

The  first  compo..ont  is  a "situation  taxonomy," 
listing  about  one  hundred  dimensions  of  a situation  that 
might  be  relevant  to  a particular  analytic  choice.  These 
dimensions  include:  the  stakes  involved  in  a decision,  the 
reaction  time  available,  and  the  clarity  with  which  options, 
probable  consequences,  and  values  are  perceived. 

The  second  component  is  an  "analysis  taxonomy," 
according  to  which  about  one  hundred  decision-analytic 
choices  can  be  located  in  an  "analytic  option  space." 
Dimensions  of  the  analytic  taxonomy  include:  how  much 
decision  analysis  is  undertaken,  how  it  is  used,  what  type 
of  model  structure  is  involved,  and  what  technique  for 
probability  assessment  or  consequence  evaluation  is  employed. 

The  third  component  is  a "performance  measure 
taxonomy,"  listing  about  thirty  measures  of  effectiveness 
which  can  characterize  the  analytic  options.  The  same 
taxonomy  can  also  be  used  to  describe  a situation  by  ex- 
pressing the  relative  importance  of  the  performance  measures 
in  the  situation.  Performance  measure  dimensions  include: 
enhanced  logical  reasoning,  cost,  speed,  convenience,  and 
facilitated  communication.  This  component  serves  as  a 
mediating  factor,  implicit  or  explicit,  in  matching  analysis 
to  situation. 


In  this  research  effort,  an  attempt  has  been 
made  to  identify  a few  important  and  plausible  matching 
generalizations  based  on  the  experience  of  practicing 
decision  analysts.  A few  analytic  options  were  selected  to 
represent  thousands  of  possibilities  and  to  facilitate 
generalizations  about  when  they  should  be  exercised  in  the 
form  of  a taxonomy  matching*  " A U.S.  decision  on  whether  to 
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export  high-technology  items  to  the  Soviet  Bloc  is  analy2ed 
by  using  the  taxonomic  matching  framework.  Other  illustrative 
material  is  also  used  throughout  the  report. 

The  second  volume  contains  five  case  applications 
of  the  framework.  The  framework  is  used  to  explain  the 
choice  of  decision-analytic  techniques  to  apply  to  cases 
involving : 


1.  the  problem  of  what  foreign  policy 
the  U.  S.  should  adopt  in  order  to 
obtain  more  oil  from  a particular 
Mideastern  country; 

2.  a decision  faced  by  the  president  / 
of  an  electrical  equipment  company 

of  whether  to  purchase  the  defense 
market  rights  to  a flight  safety 
patent; 

3.  a project  undertaken  by  the  Naval 
Electronics  Systems  Command  (NAVELEX) 
to  apply  the  "Design-to-Cost" 
concept  to  an  evaluation  of  pro- 
posed electronic  warfare  systems; 

4.  a research  and  development  pro- 
ject to  develop  tactical 
decision  aids  for  Navy  task  force 
commanders; 

5.  a study  aimed  at  predicting  NATO's 
response  to  actions  taken  by  the 
Warsaw  Pact  countries. 
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The  third  volume  contains  detailed  descriptions  of 
the  three  taxonomies  that  comprise  the  framework. 

2.1.2  Methodology  notebook  - The  second  major  task  was 
to  develop  a file  of  methodological  insights  stimulated  by 
experiences  of  prosecuting  case  studies  in  the  course  of 
other  contracts  by  DDI  staff.  Decisions  and  Designs,  Incor- 
porated, as  a major  part  of  its  activity,  conducts  decision 
analyses  of  a number  of  major  problems  in  the  public  and 
private  sectors.  The  primary  thrust  of  many  of  these  projects 
is  problem  specific,  either  to  help  an  individual  or  an 
organization  perform  their  tasks  more  effectively  or  to 
explore  the  applicability  of  decision  analysis  technology  in 
the  context  of  a specific  testbed. 

The  terms  of  the  sponsoring  contracts  do  not 
typically  call  for  direct  contributions  to  generalizable 
methodology.  The  cases,  as  developed  for  the  sponsor,  | 

however,  represent  an  indirect  contribution  to  decision  ' 

analysis  methodology  insofar  as  they  represent  an  accumu-  I 

lated  sample  (uncontrolled  and  certainly  non-random)  of 
application  experiences.  They  show,  for  example,  how  specific 
variants  of  multi-attributed  utility  models,  Bayesian  inference 
algorithms,  Paretian  models,  and  so  forth,  have  been  used  to 
address  specific  problems,  and  with  what  results. 

The  potential  for  direct  contribution  to  the 
state  of  the  art  of  decision  analysis  is  very  great,  however, 
if  additional  effort  devoted  to  that  goal  accompanies  each 
case  study.  The  cases  represent  very  rich  sources  of  insights 
and  material  relevant  to  generalized  contribution  to  the 
state  of  the  -art,  both  in  terms  of  operational  methodology 
and  more  basic  research. 

Some  such  insights  were  written  up  as  working 
papers  varying  in  length  from  two  to  sixty  pages;  others  be- 
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came  the  subject  of  specific  tasks  covered  elsewhere  in  this 
report.  Most  remained  as  notes  in  an  internal  DDI  file  to 
be  accumulated  and  uuilt  upon  until  they  reach  a point  where 
they  merit  development  as  methodological  generalizations  or 
research  proposals  through  which  to  reach  a wider  audience. 
Three  technical  reports  and  nine  working  papers  are  attributed 
to  this  task  (see  references  1,  3,  7,  8,  10,  15,  16,  17,  18, 
23,  32,  34) . 

2 . 2 Development  of  Specific  Techniques 

As  a result  of  activity  under  the  methodology  notebook 
task  above,  the  need  to  develop  special-purpose  techniques 
for  applied  decision  analysis  arose  along  with  ideas  for 
their  solutions. 

2.2.1  Modelling  subsequent  acts  - A challenge  to  con- 
ventional paradigms  for  modelling  acts  subsequent  to  the  im- 
mediate decision  was  developed  (see  references  3 & 4) . 

When  making  a current  decision,  like  choosing  an 
experiment,  a subject,  S,  will  often  take  into  account  "sub- 
sequent acts"  which  he  does  not  yet  commit  to.  Common 
practice  requires  S to  model  them  through  preposterior 
analysis,  which  treats  one  act  as  certain,  conditional  on 
the  intervening  information  modelled.  This  is  not  logically 
necessary  since  a coherent  S would  obtain  the  same  expected 
utilities  for  his  current  decision  if  he  properly  conditioned 
utility  of  any  selection  of  events  (including  subsequent 
acts) . He  could  assess  utility  marginal  on  subsequent  acts, 
or  conditional  on  subsequent  acts  treated  as  uncertain 
events.  The  preposterior  model  is  a special  case  of  the 
latter  where  conditioning  information  is  sufficiently  modelled 
to  imply  subsequent  act  probabilities  of  zero  or  one.  This 
study  argues  that  attempts  at  preposterior  modelling  are 
often  unsuccessful  and  have  critically  flawed  much  current 
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I practice  in  decision  analysis.  Simpler  approaches  such  as 

* the  "acts-as-events"  model  are  intrinsically  less  dependent 

, on  restrictive  assumptions  and  have  been  successfully  applied 

to  many  real-world  decisions. 

* This  procedure  is  illustrated  in  the  context  of 
a Navy  task  force  commander's  decision  situation,  describes 
a prototype  interactive  graphic  computer  implementation  of 
his  procedure,  and  the  results  of  a preliminary  test  and 

' evaluation  of  it  are  presented. 

2.2.2  Environmental  indices  - Techniques  for  developing 
‘ of  multi-attributed  indices  of  environmental  water  quality 

; * were  proposed  as  an  example  of  more  general  applicat.ioni;  of 

multi-attribute  utility  theory  (see  references  27  & .78). 

This  study  developed  an  approach  for  scaling 

* multi-attributed  alternatives  in  the  development  of  indices 
of  water  quality.  The  study,  conducted  in  1970-1971,  was  an 
early  application  of  techniques  developed  for  the  general 

I study  of  the  assessment  of  the  utility  of  multi-attributed 

‘ alternatives,  or  simply  multi-attribute  utility  assessment 

(MAUA) . The  work  is  recast  here  as  a case  study  with  emphasis 
on  the  steps  involved  in  the  application  of  the  procedure. 

' The  specific  question  addressed  in  this  study 

I was  whether  each  different  use  for  a water  supply  would 

require  a special  index,  or  whether  an  overall  index  would 
suffice.  The  study  facilitated  a test  of  the  feasibility  of 
applying  proposed  utility  assessment  methods  to  a complicated 
( real-world  problem.  Developing  such  water  quality  indices 

‘ requires  establishing  a suitable  mathematical  function 

defined  for  an  appropriate  set  of  parameters.  This  mathe- 
matical function  assigns  to  a complex,  multi-dimensional 
alternative  (a  sample  from  a surface  body  of  raw  water)  a 
number  which  validly  represent  the  quality  of  that  water  for 
a specific  consumer  population. 
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2 . 3 Basic  Theory 


In  the  course  of  solving  a practical  decision  problem, 
it  often  happens  that  weaknesses — or  at  least  areas  needing 
further  development — in  the  fundamental  basis  of  decision 
analysis  come  to  light.  Three  such  opportunities  were 
pursued,  and  each  case  combined  the  analysis  practitioner's 
appreciation  of  the  practical  problem  with  the  academic's 
immersion  in  the  relevant  discipline. 

2.3.1  Value  of  decision  analysis  - The  problem  of  how 
much  decision  analysis  is  worth  to  a prospective  user  is 
raised  at  least  implicitly  every  time  the  tool  is  applied. 

But  techniques  for  assigning  values  have  not  been  developed 
for  lack  of  a conceptual  framework  to  develop  them  from.  A 
Cambridge  mathematician  worked  with  DDI  staff  to  develop 
such  a framework  and  to  begin  developing  operation  algorithms 
for  practical  use  (see  references  13,  14,  35,  36). 

This  study  explored  methodological  issues, 
especially  logical  and  mathematical,  in  putting  a dollar 
value  to  a proposed  decision  analysis  exercise.  It  restricts 
attention  to  the  direct  value  of  the  analysis  in  terms  of 
impact  on  the  decision  analysed.  It  disregards  indirect 
values  (or  costs)  the  analysis  may  have,  however  important 
these  may  be  in  enhancing  organizational  processes.  Two 
alternative  valuation  instruments  are  identified,  expressed 
in  explicit  mathematical  form. 

One  approach  is  predicated  on  the  assumption 
that  there  exists  a correct  ordering  of  immediate  decision 
options  for  a given  individual  at  a given  period  in  time. 

It  can  be  loosely  defined  as  the  result  of  infinite  and 
impeccable  analytic  pains.  This  valuation  algorithm  involves 
assessing  reduction  in  "expected  irrationality  cost"  (com- 
parable to  "expected  opportunity  loss"  in  the  valuation  of 
information) . 
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The  second,  less  ambitious  approach  is  based 
directly  on  the  standard  decision-analytic  paradigm  of 
comparing  expected  utilities  with  and  without  the  proposed 
analysis.  It  requires  fewer,  if  possibly  more  elusive, 
elicitations  than  the  first  approach. 

A companion  research  effort  approaches  the 
problem  of  making  such  an  instrument  operational  through 
"after-the-fact"  valuation  of  three  completed  decision 
analysis  exercises.  In  each  case,  an  attempt  is  made  to 
gain  a hindsight  value  for  the  analysis  with  a view  to 
facilitating  "before-the-fact"  valuation  for  analogous 
decision  analysis  exercises  in  the  future. 

Latter  stages  of  the  enquiry  addressed: 

the  conceptual  mathematical  and  graphical 
explication  of  some  basic  procedures; 

an  adaptation  of  the  procedures  to  a real 
decision  analysis;  and 

the  notion  of  perfect  rationality  as  a 
reference  point  against  which  to  evaluate 
proposed  analyses. 

2.3.2  Reconciliation  of  judgmental  incoherence  - 
Different  ways  of  modelling  a problem  often  yield  very 
different  conclusions  even  when  the  model  inputs  come  from 
the  same  source;  this  gives  rise  to  a dilemma  for  the  subject 
that  the  current  status  of  decision  theory  does  not  address. 

The  dominant  paradigm  of  formal  decision  analysis 
involves  constructing  models  which  force  logical  coherence 
between  a subject's  choice  of  action  and  his  other  judgments, 
probability  and  utility.  However,  it  is  by  no  means  obvious 
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how  an  initially  incoherent  subject  should  reconcile  his 
judgments  to  form  a coherent  decision  model,  nor  how  alter- 
native ways  of  modelling  a given  probability  or  utility 
should  be  reconciled.  There  are,  in  general,  any  number  of 
ways  that  the  inputs  to  the  models  can  be  adjusted  to  yield 
coherent  systems  of  judgment.  How,  in  principle,  should  a 
single  point  in  such  reconciled  systems  be  chosen? 

A statistician  from  London  University  and  a 
psychologist  from  Hebrew  University  worked  with  DDI  staff  to 
develop  conceptual  principles  according  to  which  judgmental 
incoherence  can  be  resolved.  They  have  developed  an  approach 
whereby  a higher  order  of  judgment  is  invoked  bearing  on  the 
"precision"  of  the  subject's  original  "readings."  They 
consider  both  the  case  where  all  potential  readings  are 
taken  and  where  some  subset  of  readings  is  to  be  reconciled. 

A large  part  of  the  study  is  in  preliminary  draft  form, 
pending  the  availability  of  funding  for  completion. 

However,  a special-purpose  structure  for  use  in 
the  special  case  of  probability  assessment  has  been  developed 
to  the  point  of  publication  (reference  25) . This  paper 
investigates  the  question  of  how  to  reconcile  incoherent 
probability  assessments,  that  is,  assessments  that  are  in- 
consistent with  the  laws  of  probability.  A general  model 
for  the  analysis  of  probability  assessments  is  introduced, 
and  two  approaches  to  the  reconciliation  problem  are  illus- 
trated and  discussed.  In  the  internal  ap  roach,  one  estimates 
the  subject's  "true"  probabilities  on  the  basis  of  his 
assessments.  Least-squares  procedures  for  reconciliation 
are  developed  within  the  internal  approach.  In  the  external 
approach  an  external  observer  is  introduced,  who  updates  his 
own  coherent  probabilities  in  the  light  of  the  assessments 
made  by  the  subject. 


10 


2.3.3  Pretesting  innovation  - The  introduction  (or 
modification)  of  a management  system  in  an  organization  is 
often  preceded  by  an  effort  to  gather  data  from  which  it  can 
be  evaluated.  The  data  may  come  from  some  kind  of  experiment, 
a conceptual  simulation,  or  some  more  informal  analysis  of 
relevant  past  experience. 

Experimental  and  other  well-developed  paradigms 
often  prove  unsatisfactory,  or  at  least  incomplete,  as  a 
basis  for  validating  sequential  decisions  in  the  design  of 
systems  and  other  innovations,  for  example,  the  development 
of  operational  decision-aiding  systems  for  the  Navy.  DDI 
and  Cambridge  University  staff  have  made  an  exploratory 
attempt  to  develop  a cohesive  conceptual  framework  for 
testing  such  innovations  (see  references  15  & 16).  A paper 
on  this  subject  discusses  how  alternative  testing  procedures 
can  themselves  be  evaluated  by  paying  particular  attention 
to  analogous  testing  paradigms  in  the  more  established 
fields  of  science  and  engineering.  Decision-aiding  systems 
for  naval  command  and  control  are  used  as  an  illustrative 
case . 

2.4  Behavioral  Research 


The  choice  of  technical  approaches  for  decision  analysis 
often  involves  taking  a position  on  behavioral  issues  calling 
for  experimental  and  other  empirical  research. 

2.4.1  Group  assessment  experiments  - Choice  of  alter- 
native methods  for  eliciting  probability  assessments  from 
groups  of  individuals  calls  for  judgments  about  the  circum- 
stances under  which  one  technique  (such  as  Delphi)  outperforms 
another  (for  example,  Delbecq) . Experimental  work  to  establish 
such  generalizations  was  performed  in  collaboration  with 
Duke  University  staff  (see  reference  21) . 
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A nvunber  of  studies  have  shown  that  a consensus 
probability  distribution,  obtained  by  averaging  together  the 
assessments  of  individuals,  typically  outperforms  almost  all 
individual  probability  distributions.  The  present  study 
evaluated  several  strategies  for  improving  upon  this  averaging 
approach.  These  strategies  provide  for  some  type  of  inter judge 
interaction. 

No  between-procedure  differences  were  obtained. 

In  addition,  a re-analysis  of  data  from  a previous  study  in 
which  statistically  significant  between-procedure  differences 
were  obtained  suggests  that  these  differences  were  too  small 
to  be  of  practical  significance  to  the  applied  decision 
analyst. 

Based  on  these  results  and  a review  of  the 
relevant  literature,  two  conclusions  emerge:  (1)  subjective 
probability  distributions  can  be  substantially  improved  by 
aggregating  the  opinions  of  a group  of  experts  rather  than 
by  relying  on  a single  report,  and  (2)  from  a practice 
standpoint,  there  is  no  evidence  to  suggest  that  the  method 
used  to  aggregate  these  opinions  will  have  a substantial 
effect  on  the  quality  of  the  resulting  subjective  probability 
distribution. 

2 . 5 Case  Studies 

In  order  to  test  the  effectiveness  of  the  current 
"state-of-the-art"  and  to  suggest  and  motivate  new  lines  for 
development,  the  development  of  an  effective  decision-aiding 
technology  calls  for  applications  to  live  problems  of  national 
concern.  A number  of  such  case  studies  were  supported 
through  this  contract.  Each  case  involved  at  least  one 
other  government  agency,  either  as  a joint  sponsor  or  as  a 
source  of  funds. 


.1.. 


2.5.1  Energy  cases — Federal  Energy  Administration  - 
Three  projects  were  funded  by  the  Federal  Energy  Administra- 
tion. One  used  the  technique  of  decomposed  error  analysis 
for  making  inferences  about  domestic  energy  conservation 
behavior  based  on  survey  results.  A second  applied  person- 
alist  analysis  tools  to  the  design  and  interpretation  of 
studies  of  public  response  to  solar  heating.  A third  applied 
information  evaluation  techniques  to  the  problem  of  assigning 
priorities  to  different  types  of  research  FEA  might  undertake 
(see  references  5,  6,  19). 

2.5.2  Foreign  policy  case — U.  S.  Government  - A 
decision-analytic  comparison  of  alternative  foreign  policy 
strategies  bearing  on  Mideast  oil  (funded  by  Rome  Air 
Development  Center)  was  adapted  for  use  as  a pedagogical 
case  study  in  collaboration  with  Harvard  Business  School, 
where  it  is  used  to  train  managers  in  the  application  of 
decision  analysis  techniques  (see  reference  9) . 

2.5.3  System  acquisition — NAVCOSSACT  - Multi-attribute 
utility  analysis  was  applied  to  several  defense  procurement 
decisions  involving  system  acquisition  (see  reference  26) . 
Each  application  permitted  the  methodology  to  be  further 
refined  and  made  more  generalizable. 

In  the  spring  of  1974  NAVCOSSACT  was  confronted 
with  a problem  which  most  computer  service  organizations  are 
facing  with  increasing  frequency — computer  system  workload 
saturation. 


Although  NAVCOSSACT  agreed  to  conduct  the  competi- 
tive procurement  in  accordance  with  GSA's  requirements, 
NAVCOSSACT  wished  to  assess  the  impact  of  such  a procurement 
upon  the  costs  and  level  of  services  which  it  provides  to 
its  users  and  to  evaluate  ultimately  the  desirability  of 
that  course  of  action. 
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This  study  developed  a formal  methodology,  a 
form  of  multi-attribute  utility  analysis,  for  conducting  the 
evaluation,  with  features  making  it  generally  applicable  for 
a wide  range  of  procurement  applications. 

2.5.4  System  acquisition — PMTC  - The  purpose  of  this 
work  has  been  to  develop  and  implement  a computerized  Pilot 
Task  Inventory  appropriately  structured  and  applied  to  the 
fighter  version  of  the  F18  aircraft.  In  doing  so,  a general 
methodology,  applicable  to  other  system  acquisition  problems, 
was  developed  (see  references  29,  30,  and  31). 

The  general  approach  was  one  of  developing  a 
Pilot  Task  Inventory  by  hierarchically  decomposing  the  F18 
pilot's  tasks  into  clusters  based  upon  mission,  mission 
phase,  mission  sub-phase,  pilot  role,  and  utilized  sub- 
system. 


Two  major  problems  involved  in  such  an  effort 
were:  1)  the  development  of  a valid,  complete,  computerized 
inventory,  appropriately  structured  and  readily  usable  by 
test  pilots  and  2)  the  development  of  a valid  rating  scale 
for  use  in  rating  the  air  system  with  respect  to  suitability 
for  task  accomplishment.  The  problems  were  solved  by  using 
multi-attribute  utility  assessment  and  a conjoint  measurement 
rating  procedure,  respectively. 

Special  statistical  analyses  were  devised  which 
performed  the  following  functions:  1)  pinpointed  areas  of 
serious  rater  discrepancy;  2)  pinpointed  important  F18 
system  deficiencies;  3)  pinpointed  important  F18  system 
strengths.  It  was  also  possible  to  compare  any  two  sets  of 
individual  pilot's  ratings  for  important  discrepancies, 
generate  a mean  set  of  ratings,  and  perform  sensitivity 
analyses  on  the  weights. 


The  MAUA  assessment  with  its  accompanying  special 
statistics  was  shown  to  be  of  potentially  great  value  in 
evaluating  the  F18  and  other  major  aircraft  systems  during 
their  test  and  evaluation  phase. 
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3.0  CONCLUSIONS  AND  RECOMMENDATIONS 


The  general  research  process  represented  by  this  contract 
consisted  of  taking  ongoing  experience  in  the  practice  of 
decision  analysis  as  a starting  point  and  using  it  to  generate 
methodological  generalizations  and  to  stimulate  further 
research  of  both  a basic  and  applied  nature.  The  object  was 
to  contribute  to  improved  man-machine  interface  technologies. 

Whether  the  specific  technical  and  theoretical  develop- 
ments reported  under  this  contract  will  have  a major  impact 
on  the  "state  of  the  art"  remains  to  be  seen,  since  publication 
in  the  technical  literature  has  been  largely  concentrated  in 
the  last  six  months  of  the  contract.  If  the  authors'  conten- 
tions are  borne  out,  at  least  one  task  (modelling  subsequent 
acts)  will  have  a major  impact  on  the  practice  of  decision 
analysis,  and  two  will  have  a major  impact  on  the  theoretical 
formulation  of  the  field  (value  of  analysis  and  reconciliation 
of  judgmental  incoherence) . 

A further  task  (taxonomy  matching)  may  lay  the  groundwork 
for  the  systematic  codification  of  the  "state  of  the  art"  of 
decision  analysis — or  parts  of  it.  If  the  framework  proposed 
recoinmends  itself  to  other  researchers  and  practitioners  of 
decision  analysis,  their  accumulated  experience  in  the  field 
may  be  used  to  help  rapidly  and  efficiently  select  analytic 
approaches  for  particular  situations. 

Most  of  the  tasks  suggest  avenues  for  additional  produc- 
tive research,  not  necessarily  by  the  same  investigators  or 
research  organization.  The  modelling  of  subsequent  acts, 
the  valuation  of  decision  analysis,  and  pre-testing  innovation 
are  all  topics  which  have  been  developed  and  disseminated  to 
the  point  where  other  researchers  can  pursue  them  independently 
of  the  original  researchers.  All  three  have  been  published 
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in  the  professional  literature.  Taxonomy  matching  could  be 
pursued  independently,  though  findings  on  this  task  have  not 
been  widely  distributed  nor,  in  their  current  mode  of  presen- 
tation, are  they  easily  accessible  to  researchers  who  have 
not  been  intimately  involved  in  the  project  to  date. 

The  task  on  the  reconciliation  of  judgmental  incoherence 
stands  in  most  urgent  need  of  additional  effort  by  the 
original  investigators.  Although  a report  on  a special 
aspect  of  the  problem  addressed  has  been  prepared  and  will 
probably  be  published  in  the  professional  literature,  much 
of  the  most  general  and  fundamental  material  generated  on 
this  contract  exists  only  in  the  form  of  voluminous  but 
unpublished  notes.  It  is  conceivable  that  the  synthesis, 
development,  and  publication  of  this  material  would  represent 
a major  breakthrough  for  the  foundations  of  decision  theory. 
This  might  pave  the  way  for  researchers  to  develop  further 
special-purpose  monographs. 

However,  in  our  opinion,  the  most  important  further 
research  to  be  done  by  an  organization  such  as  DDI  is  to 
continue  to  generate  entries  for  the  "Methodology  Notebook" 
from  which  much  of  our  research  and  that  of  others  has 
derived.  There  are  a number  of  academic  and  other  research 
organizations  well  qualified  to  pursue  intensive  theoretical, 
and  in  many  cases  technical,  development  (though  it  might  be 
argued  that  DDI's  close  contact  with  live  application  of  the 
technology  contributes  a special  qualification  to  the  latter) . 
However,  there  are  exceedingly  few  other  orgemizations  where 
the  heavy  ongoing  interaction  between  technique  and  problems 
generates  such  an  abundance  of  methodological  sparks  which, 
if  immediately  applied  to  reflective  tinder,  may  set  off 
major  conflagrations  of  methodological  innovation.  We 
believe  it  is  an  essential  ingredient  for  the  balanced  and 
practially  oriented  development  of  the  "state  of  the  art" 
that  some  analytic  resources  at  least  be  devoted  to  such 
sparking  activity.  i 
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APPENDIX 
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NOTES  ON  UNPUBLISHED  WORKING  PAPERS 


! Brown,  R.V.,  The  Credibility  of  Estimates.  (December  1977). 

Executives  and  others  are  frequently  misled  by  conven- 
tional measures  of  research  accuracy,  usually  in  the  direction 
I of  overconfidence.  The  author  proposes  a research  appraisal 

tool  which  can  be  used  without  technical  expertise  to  plan 
research  or  to  assess  the  probable  value  of  variables  of 
direct  interest. 

» 

Brown,  R.V.,  V.N.  Campbell,  and  P.H.  Feuerwerger,  Value 
of  Eliminating  Uncertainty  Bearing  on  Ride-Sharing  Options : 

I A National  Analysis.  Decisions  and  Designs,  Incorporated, 

September  1977.  (Sponsored  by  ONR  and  the  Federal  Energy 
Administration) . 

I Predicting  the  value  of  information  sought  by  research 

before  the  research  is  undertaken  leads  to  better  funding 
decisions.  Here,  possible  avenues  for  research  in  the  area 
of  ride-sharing  are  investigated  in  a "quick-and-dirty " 

• attempt  to  predict  their  likely  value  by  using  the  value  of 

information  technology  of  decision  analysis. 


I Brown,  R.V.,  V.N.  Campbell,  and  D.J.  Repici,  Analysis  of 

Residential  Fuel  Conservation  Behavior;  Memorandum  of 
Findings,  Appendices,  Charts.  Decisions  and  Designs,  Incor- 
porated, October  1977.  (Sponsored  jointly  by  ONR  and  the 
I Federal  Energy  Administration) . 
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From  an  analysis  of  consumer  surveys  and  data  from 
trade  and  industry,  DDI  has  derived  probabilistic  estimates 
of  temperature  control  and  insulation  responses  to  energy 
shortage  in  U.S.  homes. 

The  data  collection  and  analysis  was  conducted  over  a 
three-week  period  and  included: 

Analyzing  of  a Gallup  home  survey  of  1,000  house- 
holds nationwide; 

Estimating  the  reporting,  non-response  and  other 
biases  and  errors  in  the  Gallup  survey;  and 

Deriving  independent  estimates  of  the  same  target 
quantities  based  on  a variety  of  sources:  statistical, 
trade,  survey  and  power  utilities. 

The  sources  of  error  in  each  approach  were  quantita- 
tively assessed  and  combined  using  the  technique  of  decomposed 
error  assessment  (DEA) , and  the  estimates  were  pooled  to 
derive  single  composite  estimates  and  probabilistic  margins 
of  error  for  each  target  quantity. 


Brown  R.V.  and  D.H.  Gustafson.  Evaluating  Decision  Analysis 
Research  Proposals:  A Case-Illustrated  Resource  Allocation 
Procedure  (WP-6) . June  1977. 

Decisions  on  how  and  where  to  spend  research  and  develop- 
ment funds  have  long  resisted  systematic  analysis.  This 
paper  describes  a decision-analytic  approach  to  the  problems 
of  comparing  alternative  research  programs  and  of  assigning 
research  dollars  among  various  projects  or  topics.  The 
objective  is  to  provide  a workable  tool  that  cah  help  the 
manager  of  a funding  agency  to  evaluate  research  proposals. 


to  assign  his  available  resources  as  decisions  have  to  be 
made  throughout  the  year,  and  to  plan  complete  agency  programs. 

A method  is  described,  grounded  in  multi-attribute 
decision  analysis  and  is  illustrated  by  using  the  hypothetical 
example  of  a defense  research  budget  for  decision  analysis 
research.  An  interactive  computer  program  carries  out  the 
successive  calculations. 


Brown,  R.V.,  J.E.  Selvidge,  and  J.W.  Ulvila,  A Classification 
of  Research  Topics  in  Decision  Analysis  (WPS).  June  1977. 

It  is  often  convenient  to  characterize  actual  or  proposed 
research  in  a field  like  decision  analysis  according  to  a 
standardized  set  of  categories  and  with  a standardized 
terminology.  An  appropriate  and  clearly  defined  classifica- 
tion permits  concise  and  precise  communication  of  research 
activity  among  researchers  and  research  users  (such  as 
decision  analysts),  for  example,  through  catalogues,  bibliog- 
raphies, libraries,  review,  and  technical  papers. 

The  classification  scheme  proposed  is  one  way  of  grouping 
actual  and  potential  research  activities  in  the  field  of 
decision  analysis.  The  perspective  taken  in  establishing 
the  categories  is  that  of  an  applied  decision  analyst  and 
stresses  the  sorts  of  questions  such  an  analyst  would  like 
to  have  answered  by  the  research  community.  Consequently, 
the  degree  of  detail  in  any  category  depends  more  upon  the 
interests  and  priorities  of  the  practitioner  than  the  volume 
of  past  and  current  research.  For  example,  "inference 
processes  of  individuals"  is  only  one  of  thirty  subtopics, 
although  this  topic  certainly  accounts  for  more  than  one- 
thirtieth  of  the  published  research  in  decision  analysis. 


Buede , D.M.,  Choosing  the  Most  Beneficial  Procurements 
with  a Fixed  Budget  (WP-1) . June  1977. 

Decision  analysts  are  often  tasked  to  determine  the 
relative  value  of  several  alternatives  that  receive  their 
value  as  the  result  of  a sequence  of  events.  This  sequence 
constitutes  a dynamic  process  which  provides  the  environment 
for  the  decision.  The  alternatives  of  the  decision  maker 
describe  his  ability  to  affect  the  outcomes  of  the  dynamic 
process . 

In  lieu  of  a major  breakthrough  in  the  modelling  of 
dynamic  processes,  this  paper  discusses  how  to  improve  our 
t ability  to  construct  and  use  static  models. 

Buede,  D.M.,  Evaluating  Dynamic  Decisions  with  Static  Models 
I {WP-2) . June  1977. 

A procedure  for  eliciting  a relative  benefit  scale  for 
a set  of  possible  procurements  is  described  here.  This 
• benefit  scale  is  a cardinal  one  but  does  not  address  the 

question  of  absolute  benefit  because  the  decision  focuses  on 
how  to  spend  a given  amount  of  money  and  not  how  much  money 
to  spend.  The  purpose  of  eliciting  this  benefit  scale  was 
I to  perform  a cost-benefit  analysis  on  the  choice  of  procure- 

ments . 

I Keelin,  T.W.,  III,  Risk  Preference  for  the  United  States 

(WP-3).  June  1977. 

Many  of  the  important  decisions  we  face  today  involve 
( the  prospect  of  very  serious  outcomes.  Most  dramatic, 

perhaps,  are  those  involving  war:  even  when  the  probability 
of  war  is  small,  its  occurrence  could  be  devastating.  It  is 


A- 4 


therefore  essential  that  decisions  involving  war,  or  any 
I other  serious  outcome,  be  made  on  the  basis  of  thoughtful, 

intelligent  analysis. 

When  high  stakes  are  involved,  the  important  risks  in 
dollars  and  lives  must  be  carefully  addressed.  To  do  so 
explicitly  requires  establishing  a risk  attitude  which  can 
be  communicated,  understood,  and  applied  in  decision  analysis. 
The  purpose  of  this  paper  is  to  derive  a utility  function 
I for  expressing  a risk  attitude  appropriate  to  the  United 

States  as  a decision  maker. 

I Selvidge,  J.E.,  Assessment  of  Multi-Attributed  Value 

Functions  (WP-4).  June  1977. 

Two  methods  of  evaluating  decision  outcomes  for  multi - 
I attributed  settings  are  contrasted  and  compared.  The 

methods  are  the  classical  utility  measure  (under  risk)  of 
Von  Neumann  and  Morgens tern,  and  the  "Regret"  assessment 
method  frequently  used  by  DDI  in  problems  having  a military 
I setting.  (See,  for  example,  the  "Warsaw  Pact  Attack"  problem 

in  Selvidge,  Rapid  Screening  of  Decision  Options,  DDI  Report 
#76-12)  . 

I 
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